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Abstract — It is shown that a minimal upper estimate for the 
error probability of the formal neuron, when the latter is used as 
a restoring (decision) element, can be obtained by the Laplace 
transform of the convolution of functions as well as by means of 
the generating function of the factorial moment of the sum of 
independent random variables. It is proved that in both cases the 
obtained minimal upper estimates are absolutely identical. 

Index Terms — generating function, probability of signal 
restoration error, restoring neuron, upper estimate. 

I. Introduction 

Let us consider the formal neuron, to the inputs of which 
different versions X 1 ,X 2 ,...,X n ,X n+1 of one and the same 
random binary signal X arrive via the binary channels 
B 1 ,B 1 ,...,B n ,B ll+1 with different error probabilities 

q : u = 1 , 7 ? + lj , and the neuron must restore the correct input 

signal X or, in other words, make a decision Y using the 
versions X l ,X 2 ,...,X n ,X n+1 . When the binary signal X 

arrives at the inputs of the restoring element via the channels 
of equal reliability, the decision-making, in which some value 
prevails among the signal versions, i.e. the decision-making 
by the majority principle, was for the first time described by 
J. von Neumann [1], and later V. I. Varshavski [2] 
generalized this principle to redundant analog systems. 

In the case of input channels with different reliabilities, 
adaptation of the formal neuron is needed in order to restore 
the correct signal. Adaptation is interpreted as the control 

process of weights a i (i = l,n+\\ of the neuron inputs, 
which makes these weights match the current probabilities 
q j (i = l,« + lj of the input channels. The purpose of this 

control is to make inputs of high reliability to exert more 
influence on decision-making (i.e. on the restoration of the 
correct signal) as compared with inputs of low reliability. 
Restoration is carried out by vote-weighting by the relation 
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Y = sgn 


where 


E a ' x ‘ 

i=i 

/7 + 1 

z = 2>,.x,. 


= sgnZ , 


( 1 ) 


( 2 ) 


Both the input signal X and its versions X i (i = 1, n) are 
considered as binary random variables coded by the logical 
values (+ 1 ) and (— 1 ) . It is formally assumed that the 
threshold 0 of the restoring neuron is introduced into 
consideration by means of the identity 0 = a u+l , where 

(— oo < a ll+l < oo ) and the signal X n+l = — 1. The main 
point of this formalism is that the signal X ll+l = — 1 is 
dumped from some imaginary binary input B n+l for any value 


of the input signal X , whereas the value q n+l is the a priori 
probability of occurrence of the signal X = +1 or, which is 
the same, the error probability of the channel B n+X . Quite a 

vast literature [3]-[7] is dedicated to threshold logic which 
takes into consideration the varying reliability of channels, 
but in this paper we express our viewpoint in the spirit of the 
ideas of W. Pierce [ 8 ]. 

Let us further assume that 

-1 ifZ<0 

sgnZ= . (3) 

+ 1 ifZ >0 

When Z = 0 , the solution Y at the output of the restoring 
formal neuron has the form +1 according to (3). The 
probability that the restored value Y of the signal X is not 
correct is expressed by the formula 

Q = Prob {Y * X} = Prob {77 < 0} . (4) 

Here 77 = XZ is a discrete random variable with 
probability distribution density f(y) . This variable is the 
sum of independent discrete variables //, = « XX , , and the 
function /i(v,.) describes the probability distribution density 
of individual summands i] i . For the realizations of random 


variables 7) and Tj i we introduce the symbols v and v ; , 
respectively. 

It is easy to observe that the variable 77 , takes the values 
+a j and - a with probabilities 1 — q t and q t , respectively. 
Therefore, if we use the Dirac delta function S(t ) , then the 
probability density f] ( v t ) can be represented as follows 
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f (v, ) = (1 - q, )6(v,. - a, ) + qP(v, + a , ) 


III 

'+T 

T 

v, =+a n — a i 

(5) 

i = 1, n + 1 


y = i, r +l 

Such formalism is completely justified and frequently used 

where 

due to the following two properties of the delta-function 

8{t) > 0, Vt e Rj 

q k =■ 

J ^S(t)dt = 1 J' 


for any k [k = 1, n + 1 j 


However / ( '(v ; ) can also be represented as 

r / \ (fl —v )/2a l /-i \(a +v-)/2a, 

f(y i ) = qp (1 -t?,)' 

V,. = +<7,. , - <7,. 


i = 1, n + 1 




q k if v k =-K) 
if h = +(«*) 


Thus formula (9) can also be written in the form 

Q=Y;Qj = Yj ( h' < l* Vn-q n+ 1 . ( 10 ) 

v<0 v<0 

(6) which is more adequate for cognitive perception and practical 
realization. 


The random variable q is the sum of independent discrete 
random variables ij j . Its distribution density /(v) can be 
defined in the form of convolution of probability distribution 
densities of summands f] (v ( . ) : 


/(v)= * ,/I(v,), (7) 

where * (superposition of the addition and multiplication 
signs) is the convolution symbol. 

It is obvious that in view of formula (7) the error 
probability at the decision element output can be written in 
two equivalent forms 

» ° »+i 

Q = Prob(v < 0) = J f(v)dv = J * f(v,)dv = 

( 8 ) 

= j *[(!-?, )<5(v, ~a l ) + q l d(v i +a l )]dv 


and 


e=E/( v )=E"*/i(b)= 

v<0 v<0 1 

= Y "** \q\ a ‘ ~ v ‘ ),2a ‘ ■ (i - q, ) ( “' +v ' ),2a ‘ } 


(9) 


where the probability distribution density j) (v ( . ) is defined by 


(5) in the first case and by (6) in the second case. Integration 
or summation in both cases is carried out continuously or 
discretely over all negative values of the variable v . 
Formulas (8) and (9) give an exact value of the error 
probability of restoration of a binary signal by the formal 
neuron. 

Note that for practical calculations, formula (9) can be 
written in a more convenient form. Indeed, the complete 
number of discrete values of the variable v is 2” +1 since 


v = a x +a 2 -\ \-a„+a n+l , 

where a i is equal either to +(a,-) or to -(a,), whereas the 
proper sign of the weight a t is meant to be within the round 
brackets. 

By formula (9), to each discrete value of the sum V there 
corresponds the term Q ( j = 1, 2' ,+1 ) which is the product of 
(« + l) co-factors of the form q k or (1 — q k ). 

In particular 


II. FINDING A MINIMAL UPPER ESTIMATE BY THE 
FIRST METHOD 

From the expression 

0 

Q = Prob (/7 < 0) = J f(v)dv 

—oo 

it follows that for a real positive number .v (s > 0) 

0 00 

e<je-"f(v)dv<Je-"f(v)dv. 

—oo —co 

But the left-hand part of this inequality is the Laplace 
transform of the function f (v) 

co 

£[/(v)] = j e~ sv f(v)dv, 

—00 

where £ is the Laplace transform operator. 

Therefore 

Q<2[f{v)]. (11) 

The random value q with realizations v is the sum of 
independent random variables q i having realizations v, . In 
that case, as is known, the Laplace transform for the 
convolution / (v) of functions j] (v, ) is equal to the product 
of Laplace transforms of convoluted functions: 

n+i 

£[/(v)]=n£[./i(v,)]. 

1=1 

The latter implies that 

n+l 

Q<X[Z[f t (y t )l (12) 

1=1 

By expression (5) for functions _/J(v,.) and the Laplace 
transform definition, we obtain 

oo 

£W(v,)]= \e~ s ”f i (y i )dv i = 

—CO 

CO 

= | e~ sv ‘ [(1-<7,)<5(v ; -aJ + qtSty +a i )\dv i 

—oo 

Using this expression in formula (12) we have 

n + 1 °° 

2^n j e_SV ' [( 1_ 9,)^(v, -a i ) + q l S(v l +a i )\dv l . (13) 

1—1 —oo 

Here we should make use of one more property of the Dirac 
delta function 

co 

J g(t)S(t-t n )dt = g(t 0 ). 
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With this property taken into account, from formula (13) 
we obtain 


^n[(w, ) e "“ ,s + y< e+a ' s ]• 


(14) 


g, 


Here s , as mentioned above, is an arbitrary real positive 
number. Before we continue simplifying the right-hand part of 
inequality (14), we have to define a value of s for which 
expression (14) gives a minimal upper estimate. 

Passing to the natural logarithm of inequality (14) we come 
to the expression 

\nQ<f^\n[(\- qi )e- a ‘ s +qi e^]. 

i = 1 

Let us define here partial derivatives with respect to 
arguments a i by using the elementary fact that 

/ = ^ = /'(*)-e /w 

ax 

if y = e f(x) , and also the fact that —In f (x) = f'(x) — - — . 

dx ‘ f (x) 

Hence we obtain 


i -q, 

i ~q, 


= exp 


= exp 


In 


In 


g, 

!-g, 


!-g, 


= exp 


= exp 




1 . 1 - g, 

+ — In — 


Besides, we denote 
1 
2 


C 


1 — g/ 




a - In 

v g/ 


= A. 


X , -X 

e ' +e ^ 


Then we have 

a t Oj 

g,.e 2 +(\-q i )e 2 = 2^.(l-g,) . 

The second co-factor in the right-hand part of this 
expression is the hypebolic cosine of the argument 2 : 


e ** +e 


= eh a, . 


Therefore 


Sin Q 


*1; 


1 


{sq^ -s{\-q i )e~ a,s l 


5a,. w [(1 - g, )e~ a,s + q t e +a,s ] 

For the right-hand part of this inequality to be equal to 
zero, it suffices that the following condition be fulfilled 

sq t e* v -sQ-q^e* = 0, 


q t e 2 + (1 -q,)e 2 = 2^/<7,(l-<7,) • eh /l, = 
= 2^/g, (1 - g, ) • eh 


, 1 — g, ^ 
a, - In 

g, 


v v 

Finally, for estimate (14) we can write 


whence it follows that 

n + 1 

1=1 


fa, In 1 

+2 a,s 1 — g; 

2-y/g, (!- g, ) ■ ch 

g, 

e ' - 

g, 

2 

or, which is the same, 


__ 

v y. 


! — g, 

as = —In 


2 q t 

If the weights a, of the neuron inputs are put into 
correspondence with error probabilities q of these inputs by 
the relations 

a, =ln- — — , (15) 

g, 

then the sought value of s will be 

*4 (i6) 

Using equality (16) in formula (14), we obtain a minimal 
upper estimate for the error probability Q of the restoring 
neuron. Indeed, for the right-hand part of expression (14) the 
following chain of identical transforms is valid: 

a j Oj 

A , a.e T +(l-q.)e~ T 

g,e 2 +(1 -q,)e =2^q i (l-q i ) ' — = 

2 vgiU-g,) 


For the error probability Q , the right-hand part of the 
above inequality is the upper estimate Q : 


n + 1 


e*=n 


i = 1 




4 , 1 -<7, x 

a i - In 


- 2^g, (l-g, ) ■ — 


!-g, 


•e- + 




The minimum ^) nijn of this upper estimate (9 is equal to 
QL = f] [2Vg, a - g, ) ] = 2” +1 • f[ (! ■ - 9, ) ] • (17) 

It is attained when for the zero argument the hyperbolic 
cosine attains the minimum equal to 1 . 

This estimate confirms in a certain sense the advantage of 
the choice of weights of the restoring neuron in compliance 
with the error probabilities of input signals according to the 
following relations 

1 -gf 


1-g, 

g; 


a, = In - 


g, 


i = 1, n + 1 


Let us take into account here that 
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III. OBTAINING A MINIMAL UPPER ESTIMATE BY 
THE SECOND METHOD 

Simulteously, for the probability Q it is useful to obtain a 
minimal upper estimate in the closed analytic form by one 
more new approach. 

As is known [9], the generating function 7^ (.S') of the 
factorial moment of the sum 7/ of independent random 
variables // is equal to the product of generating functions 
7 V (5) of the factorial moments of individual summands, i.e. 


where 

Q* =( 1 -^K+— 

w i 

and, along with this, 

w i = S a ‘ , 0 <w i < co, (7 = 1, n + 1). (25) 

Now we can find the minimum Q* m of expression (24) and 
the value w oi of w : will attach a minimum to the upper 
estimate of the error probability Q + of the restoring neuron. 
For this, we use the conditions 


^)=riM 5 )’ 

i = 1 

where 

y v (S)=M[s"] = X*r-/(v), 

V 

r (i (5)=M[^] = ^- J ((v i ) 


(18) 

^ + =0 


dw i 


i = 1, n + 1 

( 19) Hence 


(20) 


(26) 


i = 1, n + 1 

Here M is the mathematical expectation symbol and 5 
is an arbitrary complex number for which series (19) and (20) 
exist on some segment of the real axis containing the point 
5 = 1. 

Since in relation (20), summation is carried out on the set 
of two possible values +a r and —a j of the variable v . , using 
(6) we have 

y V! (S) = 0- q ,)S“- +q t S- a ‘ 
i = 1, n + 1 

The substitution of (21) into relation (18) gives 

nW = n[(i-<?,)'^+^‘]- 

J=1 



If (26) is substituted into expression (24), then by the 
second method for a minimal upper estimate of the error 
probability of the restoring neuron we obtain the relation 

/? + ! r 

(27) 

which coincides with result (17) obtained by the first method. 
The weights a I (i = l,n + Y) which match the error 

probabilities q t I i = 1, n + 1 1 are defined from relations (26) 


with notation (25) taken into account: 

-- 1 -i„-» 


2 In 5 1-g, 


i = 1, n + 1 


When v < 0 , the value S' satisfies the condition 


if of course 



0 < 5 < 1. (22) 

Let us assume that inequality (22) is fulfilled. Then the 
following relation is valid: 

0 = Z/(v)<£s''./(v). 

v<0 v<0 


Since the value 5 satisfies condition (22), we have In 5 < 0 
and therefore 


a t = K ■ In 


1 -q, 

Q, \, 


i = 1, n + 1 


(28) 


where 


K = 


1 

2|ln5| . . 


(29) 


Since every summand 5 V ■ / (v) is non-negative, we have the 
inequality 

YSV(v)<ZS v /(V). 

v<0 v 

Therefore 

Q<r v (S). (23) 

The right-hand part of this expression can be taken as the 
upper estimate Q + of the error probability Q of the restoring 
neuron 


0 + = fl [(1 - q, )5“' + q ; S^ a ' ] . 

1= 1 

ition is easily re 

n + 1 n + 1 

e‘-ne*-n 


The latter relation is easily rewritten in the equivalent form 

n + 1 «+ 1 


(l-?i)W/+ — 

W; 


(24) 


0 < K < co 

Thus, the weights « (i = 1, n + 1) , which are consistent with 

the error probabilities q t = 1, « + lj and attach a minimum 

to the upper estimate of the error probability of the restoring 
neuron, are defined to within the a general positive factor K . 


IV. Conclusion 


A minimal upper estimate of the error probability of the 
restoring formal neuron is defined by formula (17) or, which 
is the same, by formula (27). In both cases the result can be 
written in the form 


f n+i 


Qmm = exp 


V 1=1 


(30) 
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where 


Aq,) = 


In [2^,(1 -<,,)] 


(31) 


In view of relations (31) confirming the non-negativity of 
the values A(q,) , formula (30) implies that an increase of 
the number n of inputs of the formal decision neuron bings 
about a monotone decrease of the minimal upper estimate of 
the error probability of restoration of the binary signal Q* r:m 


by the exponential law if, certainly, the error probabilities 


= 1 , 77 + lj at these inputs are not equal to — when the 


minimal upper estimate of the error probability Q~ in is equal 
to 1. 

This result is demonstrates an essenatial inner connection 
with Shannon’s theorem [10]. According to this theorem, the 
number of messages of length n (duration r ) composed of 
individual symbols - both in the absence and in the presence 
of fixed and probabilistic constraints (in the latter case it is 
assumed that the source is ergodic) - grows by the 
asymptotically exponential law as n (or r ) increases. In 
particular we understand this connection as follows: as the 
number n of inputs of the restoring formal neuron increases, 
the initial information to be used in making the decision Y 
increases by the exponential law if there are a number of 
possible versions of the input signal, while the minimal upper 
estimate QY m of the probability Q that the made decision is 
erroneous decreases by the same exponential law. 


References 

[1] J. Neumann, ’’Probabilistic logics and synthesis of reliable organisms 
from unreliable components", in Automata Studies: Annals of Mathematics 
Studies , no. 34 (in Russian, translated from English), Moscow: Publishing 
House for foreign literature ILL, 1956, pp. 68-139. 

[2] I. Varshavsky "Functional Possibilities and Synthesis of Threshold 
Elements" (in Russian), Dokl. AN SSSR, vol. 139, no. 5, pp. 1071-1074, 
1961. 

[3] M. Dertouzos Threshold logic (in Russian, translated from English), 
Moscow: Mir, 1966. 

[4] C. R. Lageweg, S. D. Cotofana, S. Vassiliadis, "A full adder 
implementation using set based linear threshold gates", in Proc. 9th IEEE 
International Conference on Electronics, Circuits and Systems , New York, 
2002, pp. 665-669. 

[5] V. Beiu, J. M. Quintana, M. J. Avedillo, "VLSI Implementation of 
threshold Logic: A comprehensive Survey", IEEE Trans. Neural Networks , 
vol.. 14, pp. 1217-1243, May 2003. 

[6] R. Zhang, P. Gupta, L. Zhong, N. K. Jha, "Threshold Network 
Synthesis and Optimization and Its Application to Nanotechnologies", IEEE 
Transaction On Computer-Aided Design Of Integrated Circuits And 
Systems , vol. 24, pp. 107-118, January 2005. 

[7] T. Gowda, S. Vmdhula, N. Kulkami, K. Berezowski, "Identification 
of Threshold Functions and Synthesis of Threshold Networks", IEEE 
Transactions on Computer-Aided Design Of Integrated Circuits And 
Systems , vol. 30, pp. 665-677 , May 2011. 

[8] W. Pierce, Failure-tolerant computer design (in Russian, translated 
from English), Moscow: Mir, 1968. 

[9] G. A. Korn, T. M. Korn, Mathematical handbook for scientists and 
engineers (in Russian, translated from English), Moscow: Nauka, 1977, p. 
832 

[10] S. Goldman, Information theory (in Russian, translated from 
English), Moscow: Publishing House for foreign literature IIL, 1957. 


Archil I. Prangishvili was bom April 13, 
1961, in Tbilisi (Georgia). During 1978-1983, he was the student of Faculty 
of Automatics and Computer Engineering at Georgian Polytechnic Institute. 
During 1983-1987, he was the Post graduated student at Georgian 
Polytechnic Institute. Currently, he is the Doctor of Technical Sciences, Full 
professor at Faculty of Informatics and Control Systems of Georgian 
Technical University, full member (academician) of the Georgian National 
Academy of Sciences, President of the Georgian Engineering Academy, 
Member of the International Engineering Academy and the International 
Academy of Informatization of the UN (United Nations), Rector of 
Georgian Technical University. Archil I. Prangishvili is the expert in the 
field of Computer Science, Applied Expert Systems, Artificial Intelligence 
and Control Systems, head of Editorial Board of the Journal “Automated 
Control Systems” and member of the Editorial Board of the Georgian 
Electronic Scientific Journal (GESJ): Computer Science and 
Telecommunications. Number of published works - more than 120. 
including 7 monographs and 5 textbooks. 


Oleg M. Namicheishvili received the 
Master of Science (Radio physics) from Ivane Javakhishvili Tbilisi State 
University in 1965. During 1965-1968, he studied at the Graduate School of 
Tbilisi State University and he was a trainee at the Department of theory of 
probability in Moscow State University. In 1968 he received the degree of 
PhD (Engineering Cybernetics and Information Theory) and in 1989 he was 
doing D. Sc. (Tech.) in Georgian Technical University. Over a long period of 
time Oleg Namicheishvili was an internship at CNET (Centre National 
d’ Etudes des Telecommunications, France, Lannion) - a French research 
laboratory in telecommunications. He was now working in Faculty of 
Informatics and Control Systems of Georgian Technical University as Full 
Professor and ALT (Accelerated Life Tests) Project Director. He has a 
teaching job and his research affected the reliability of information systems. 
He has published more than 150 research papers in various journals. He is a 
full member (academician) of the Georgian Engineering Academy and 
Deputy Editor-in-Chief of the Georgian Electronic Scientific Journal 
(GESJ): Computer Science and Telecommunications. 


Maia A. Gogiashvili was bom September 
23, 1971 in Tbilisi (Georgia). In 1994 she graduated Ivane Javakhishvili 
Tbilisi State University, faculty of Physics. Since 1994 she is teacher of 
physics and mathematics at Grigol Robakidze University. From 2005 she is 
an expert at the National Examinations Center of the Ministry of Education 
and Science of Georgia. Currently she is a doctoral candidate of St. Andrew 
the First-Called Georgian University of the Patriarchate of Georgia at 
Faculty of Information Science, Mathematics and Natural Sciences. She has 
published more than 3 research papers in various scientific journals. 





68 


www.erpublication.org 


